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ABSTRACT 

An investigation was carried out to determine the 
feasibility and relative cost of developing pictorial procedures that 
could be used in conjunction with the Lincoln Training System (LTS) 
for task simulation in the support of performance laboratory 
instruction. Results showed that the technique was economical and 
effective. The storage and data processing capabilities of the LTS 
made it possible to monitor and assess the student* s performance. It 
was also possible to record and monitor system performance in the 
same fashion^ and a scheme for "system performance assurance" was 
developed which capitalized on this capability. . (Author/PB) 
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ABSTRACT 



An investigation was carried out to determine the feasibility an(i relative 
cost of developing pictorial procedures that could be used in conjunction 
with the Lincoln Training System for task f emulation in the support of 
performance laboratory instruction. The teclirique appears to be eco- 
nomical and effective. The storage and data processing capabilities of 
the I.TS make it possible to monitor and assess t)je student's perform- 
ance . Jt would also be possible to record and monitor system perform- 
ance in the same fashion, and a scheme for "system performance 
assurance" is developed which capitalizes on this capability. 
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USE OF THE LINCOLN TRAINING SYSTEM (LTS) FOR TASK SIMULATKjN 
IN THE SUPPORT OF PERFORMANCE LABORATORY INSTRUCTION 



I. SUMMARY 

Current Air Force technical traininj^ begins with tho oeterminatioii of job perfcrrnance 

requirements and is ultimately evaluated by how effecKvely it qualifips personnel for success- 

1 

ful performance on the job (Ai^lVI f 0-2). With this focus, the ''poriormance laboratory" becomes 
a particularly important component of the instructional process. l!nfci*tunately, it i« also a 
particularly expei.cive form of instruction that often l equires the duplication of expensive equip- 
ment, the development of special trainers or simulators and, characler. jtically, a very lo^v 
student/instructor ratio. 

It appeared that the Lincoln Training System (LTS) might be cost-effective mr ta.?k simula- 
tion in support of at least some aspects of performance training. In order to investigate this 
possibility, a sequence of instruction on the calibration of a Tektronix 545A oscilloscope was 
developed with the dual aim of (a) establishing the feasibility and relative cost of developing 
pictorial procedures for student guidance and drill, and (b) providing a means for evaluati»*g 
particular features of the LTS in this application. 

It was determined that pictorial procedures could be developed relatively easily by photo- 
graphing an expert in the performance of the task and that such material could be used effec^ 
tively to support performance laboratory instruction. This technique appears to be economical 
in comparison with other means of developing and publishing performance aids. 

The .symm;?try between task trainers and proceduralized job performance aids was noted, 
and a major result of the investigation has been development of the concept of System Perform- 
ance Assurance (SPA), to be achieved through the use of pictorial procedures and an automated 
maintenance and performance monitoring system. In this context, it appears that a modified 
LTS could insure improved system performance and also support meaningful on-the-job train- 
ing for operations and maintenance personnel. • 

II. BACKGROUND 

The systems approach to instructional system development adopted by the Air Force in 
1970^ represents an orderly process of gathering and analyzing job performance requirements, 
translating these requirements into behaviorally stated learning objectives, and integrating 
instructional techniques and procedures to assure achievement of these objectives. The empha- 
sis is on the job and job performance. Knowledge of principles, the acquisition of a specialized 
vocabulary, and the ability to "verbalize" the job are regarded as secondary and are included in 
formal instruction only to the extent that it is believed that they may improve job performance. 

With this approach to instruction, it is apparent that the performance laboratory takes on a 
very central role. The reference here is to training on the specific tasks that the student will 
be expected to perform in the field. Basic skills training — soldering, welding, the use of hand 
tools and simple instruments are elements in the performance laboratory, but our focus here 
is on more complex jobs such as the tuning, calibration, trouble shooting, and repair of equip- 
ment or exercises in operational procedures using mock-ups, simulators, or actual field sys- 
tems. In most such cases, the emphasis is on leaming a sequence of behavior, it being assumed 



that the individual actions - meter reading, knob turning, cover removal, probe insertion, etc. - 
are already in the student's repertoire. 

These procedural sequences will generally not be fixed, and present actions will be depend- 
ent on previous actions and the outcome of prior tests or measurements. The most difficult 
aspect of this sort of learning is the establishment and remembering of the discriminative clues 
that determine the specific sequence of actions that must be carried out under various circum- 
stances. In maintenance and repair tasks, the appropriate procedures will often have to be 
derived from a Technical Order and the student's knowledge of the principles of operation of the 
system at hand. Technical Orders emphasize system structure and operation; usually, they do 
not lay out procedures for fault location, part replacement, or repair, and they are not written 
for apprentice repairmen. It is therefore necessary for the instructor in a performance labora- 
tory to demonstrate procedures and then to guide and correct the individual student as he attempts 
to carry them out on his own. This is essentially a tutorial situation. Student/instructor ratios 

tend to be very low and the associated cost correspondingly high.* 

2 

It seemed apparent that the Lincoln Training System (LTS), which is an interactive system 
thai has been developed to support tutorial -like training, could be used in this context for the 
following reasons. 

(a) It has the capability of guiding the student through any step-by- step operation at the 

3 4 

student's own pace, a procedure of known effectiveness. ' 

1 

(b) The audio capability should alleviate the reading proficiency requirement that has 
been a problem with some proceduralized performance aids. 

(c) Audio instruction should facilitate performance where the student is engaged in a 
difficult manipulative task. 

(d) The LTS is designed around a pictorial display that should minimize errors of 
instruction arising from ambiguity in textual descriptions of procedures. 

(e) The branching capability of the systerts could be used for quick reference to part 
numbers, circuit diagrams, etc., or for entiy to explanatory or theoretical mate- 
rial, either automatically or imder student control. 

In this context, the system would be used in lieu of an instructor who might otherwise be 
guiding the student through the task at hand. As we have noted, programmed instruction of this 
sort has been shown to be effective, but the generation of instructional material appropriate to 
this kind of application has proved to be difficult and expensive.^ Using the LTS it would, how- 
ever, be possible and perhaps economical to use pictorial procechires generated by simply photo- 
graphing the instructor, or expert, as he carries out the Job. A pictorial aid of this sort would 
in effect put the student in direct contact with the instructor and should eliminate errors and 
ambiguities introduced by attempts to translate actions into words. 



* The present Metrology (precision measuring equipment) course at Lowry AFB is a good illus- 
tration. This is a 1290-hour (43-week) course with approximately 50% of the time devoted to per- 
formance laboratories. Students are taught to calibrate and repair a wide variety of equipment, 
most of which is expensive and fragile and therefore not duplicated in the laboratory. As a result, 
each stt^dent, or pair of students, may be working with different equipment and require separate 
supervision or instruction; 2:1 student/instructor ratios are common under these conditions. 
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Accordingly it was decided to develop a brief sequence of instruction with the dual aim of 
(a) providing a means for evaluating particular features of the LTS in the context of a perform- 
ance laboratory, and (b) establishing the feasibility and relative cost of developing pictorial pro- 
cedures for this purpose. 

III. PROCEDURE 

A subset of the calibration' procedures for a Tektronix 545A oscilloscope was chosen as a 
vehicle for demonstrating the feasibility of this approach to performance training and the gener- 
ation of proceduralized instructioi**al material. The task chosen represents 30 to 60 minutes of 
activity on the part of a highly trained technician, the length of time depending on how much 
adjustment is required. 

We began by observing an individual performing the task; whenever a change in operati'^nal 
setup was required, a requirement for a photograph was established. When test points or con- 
trols were difficult to see, a requirement for a close-up photograph was indicated. Approxi- 
mately 60 discrete photographs were thus identified (1-2 per minute of actual activity). 

A flow chart of the calibration procedure and photographic requirements was esta.>lished in 
this fashion, and the complete procedure was then performed by the instructor in stop-action. 
Photographs (identified from ;he flow chart) were taken first as Polaroid samples, and then final 
exposures v/ere made. A voice recording was made of the instructor's comments and explana- 
tions as he performed calibration tasks. 

The final calibration procedure flow chart was then assembled using the Polaroid prints for 
each frame, the text messages required for trainee response options were added, and frames of 
specific audio messages were prepared. The size of each final glossy continuous tone print was 
then specified, the intent being to use as n?T^.ch of the 8|-- X 11 -inch display area as possible for 
viewing ease. Frame notes were developed indicating arrows or other special additional marks 
wnich would be required to complete the visual displays. Audio frames were recorded, using 
the instructor's audio tape for guidance, and LTS procedures were u£2d for assembling the 
frame branching logic. 

Four standard LTS microfiche cards, containing a total of 41 audio-graphic frames, were 
produced and used for procedure debugging purposes. Approximately 10% of the graphics, text, 
audio, and logic content was found to be in error and revised microfiche were produced. The 
material was also produced in booklet form (available on request) in order to determine the 
ease with which this material could be translated between these two methoc^s of presentation. 
Only minor modifications in the audio text, which is printed on the pt^^e facing the pictorial 
material in the booklet, were required. In addition, frame branching (page location) options 
were added to the text. 

Four laboratory technicians served as trainees to determine whether thiey could calibrate 
the Tektronix 545A oscilloscope supported solely by pictorial procedures. The technicians had 
no prior experience with oscilloscope calibration procedures and had not used the required test 
equipment prior to this test. Three technicians satisfactorily calibrated the oscilloscope in 
1.75 to 2.0 hours, using the LTS, and one technician completed the task in 2.5 hours, using the 
booklet. The time required (first trial) by the trainees was 2 to 4 times that required by a 
highly experienced instructor, and we did not attempt repeated trials to determine the rate at 



which these times might improve."^ The important point is that our technicians were able to 
perform the calibration successfully the first time, without requiring the instructor to be 
present. 

As would be expected, users were able to point out a number of minor problems in the con- 
tent of the procedure. In general, these problems related to graphic displays without adequate 
pointers and audio frames with too much content, which made them difficult to remember or 
required that they be repeated several times. A final revision would require changes in approx- 
imately 20% of the frames to be responsive to these complaints. 

This observation is of interest because revision after initial trials must be anticipated in 
the development of any training or performance aid, and the ease and cost of such revision must 
be factored into any cost estimates or cost comparisons among different forms of publication. 
This is explicitly taken into account in the cost estimates for pictorial procedure development 
that are given in Appendix I. 

IV, DISCUSSION 

The work described above demonstrates, that it is relatively easy and economical to develop 
pictorial procedures that can be used to guide people through quite complex tasks with which 
they have had no previous experience. However, if these procedures are to serve as training 
aids it is necessary Iso that some provision be made to monitor and assess the student's per- 
formance. In particular, it might be desirable for the system, rather than the student, to 
determine the need for remediation and to control branching to additional exercises, further 
explanation^ etc. 

In order to accomplish this, it will be necessary for the student to input appropriate infor- 
mation^to enable the system to make such decisions- To illustrate how this might be done, con- 
sider the procedure for a step-by-step check of the oscilloscope low- volt age power supplies. 
The general strategy employed is for the student to make a voltage reading, compare this with 
a nominal value that is presented to him, and select the next frame on the basis of the difference 
between the observed and nominal values. With the LTS it would be possible, however, to have 
the student enter the observed voltage through the keyboard and have branching to an adjustment 
sequence or to the next measurement programmed within the machine, contingent on the value 
entered. 

If the procedure is considered purely as a performance guide^ there is probably little to 
choose between student or system initiated branching: On the other hand, if the student is not 



* It is our opinion that the observed initial difference in time required by the student using the 
booklet and those using the LTS would hold up if the size of the sample were increased. This 
is supported by the observation volunteered by the LTS users that the audio supported a feeling 
of confidence that the actions they took, or were about to take, were correct and safe. The one 
technician using the booklet reported a lack of confidence and a concern that pre-conditions for 
his current actions were possibly not complete. This was reflected in considerable backtrack- 
ings checking, and rec becking of prior frames. 

It is interesting to compare this reaction to the audio with similar comments from Air Force 
trainees using the LTS at Keesler AFB, In designing the LTS, a requirement was established 
for audio as a means for minimizing the ne'' \ for reading proficiency. It is increasingly appar- 
ent, however, that the audio has an important affective as well as informational content. This 
is reinforced by the unexpected observation noted here. 

There is, of course, no reason to believe that repetition would not increase student confidence 
without the audio, and we would not expect significant differences in the ultimate performance 
of this type of task by students trained with booklets or the LTS. 
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given nominal values, he is forced to carry out the necessary measurements and he is precluded 
from successfully guessing his way through a procedure.* Furthermore, such information would 
make it possible to take advantage of the memory and logical processing capacity of the computer 
to condition branching decisions on the student's cumulative performance as well as his current 
response. In the simplest case, the student could be transferred to a different sequence of 
frames when a cumulative index of' responses made in one of several areas exceeds some estab- 
lished criterion. 

The conditional branch feature has now been implemented in LTS software and has made it 
possible to simulate a wide range of problems in diagnosis and trouble shooting. The general 
approach is to present the student v/ith the gross performance {or symptoms) of the system being, 
simulated. The student then selects any one, or several, diagnostic tests which he inputs to 
the LTS. He is then informed of the results of these tests and on this basis chooses a tentative 
diagnosis and runs more tests, or makes r final diagnosis. He is then told whether or not his 
diagnosis is correct and, in either case, he receives further feedback which tells him if he has 
performed all of the pertinent tests. Treatment, or repair, is handled in a similar fashion, and 
the software permits constraints to be put on the order in which tests or treatments are carried 



Cumulative evaluation should also permit the development of more efficient pre- and post- 
tests of skill and knowledge. For instance, the student might be required to answer a series of 
questions, receiving Immediate feedback for each answer, but with the length of the test condi- 
tioned by the error rate or error type. The system would thus allow for tests which are indi- 
vidualized in both content and extent. 

When the student keys into the LTS the results of his tests and observations, it is not only 
possible to monitor his performance but alternatively to obtain a record of the performance of 
the system that is being studied. This capability suggests an application of the LTS that goes 
well beyond its function as a task trainer, and we now consider the problem of maintaining 
system performance rather than the more limited though related problems of training and pro- 
vid.ng performance aids to operations and maintenance personnel. 

<» 

V. SYSTEM PERFORMANCE ASSURANCE 

Procurement policies that have been in effect for over a decade have introduced into the. Air 
Force high technology systems that are designed to be stable « reliable, and maintainable. To 
achieve this, the Air Force has paid substantial premiums on built-in instrumentation for per- 
formance monitoring, for marginal checking systemsj modular design, and ^*throw-away" main- 
tenance, and these measures have not generally been sufficient to assure continued effective 
operation. At the time of acceptance from the vendors, these systems meet or exceed specified 
criteria; after a time in the field, maintenance costs are found to be greater than anticipated, 
MTBP is shorter than expected* and system performance is substantially below specification. 



This would not, of course, be true in carrying out as simple and well specified task as the 
voltage checks in the present example. 

t With this addition to the LTS software, it was possible to transfer to the LTS a simulation 
program, developed by the University of Kentucky Dental School for the IBM 1500 system, in- 
volving the diagnosis and treatment of endodontic dLsease. This material was selected because 
it had already been used extensively and successfully to develop skills comparable to those re- 
quired in trouble shooting and repair of complex systems. 



out if that is important.^ 
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As an example, an intensive study (Project SCOPE CREEK) of the Air Force world-wide 
communications system indicated that perfori. ance of that system — individual links as well as 
over-all — had decreased about 3 dB from 'Tkc r.ew" operation by the end of the first year of 
operation and had decreased by 17 dB at the end of 2C n.onths.^ There is reason to suspect com- 
parable degradation in other Air j 'crce systems. 

The usually suggested remedy i r probl^ ms of this sort is to provide increasing numbers 
and/or more extensively trained operations and maintenance personnel. (This has been the 
approach taken by AFCS to alleviate the deficiencies noted above.) However, this is an expensive 
and not generally applicable solution because 

(a) It is difficult to obtain and train appropriate personnel. 

(b) It is difficult to insure thai field maintenance and operational personnel adhere to 
approved maintenance procedures. 

(c) Operational and maintenance procedures should be continually updated in response 
to the accumulation of operations data and field or factory equipment modifications. 
It has proved to be extremely difficult to track this information and to insert it into 
the training-field operation cycle in a timely fashion. 

On the other hand, as mentioned above,- sophisticated engineering design has actually sim- 
plified the operation and maintenance of contemporary systems. Expert knowledge is not re- 
quired to make measurements, to record them, to adjust voltage levels, to check monitoring 
instruments, and to replace defective equipment modules. What would seem to be required is 
a set of precise, up-to-dave job guides or FPJPA' s (fully proceduralized job performance aids), 
and a monitoring system that will (a) insure that the procedures are properly carried out, and 
(b) provide a continuous record of subsystem and system performance. 

To be effective, the monitoring system must be automated and centralized to whatever ex^ 
tent is necessary to enable merging and tracking of performance data from individual units or 
subsystems in a timely fashion. (Data for large systems will generally be acquired at different 
times and from different locations.) Such records would make possible a variety of actions 
beyond the simple accumulation of failure data for logistic support. These would include revi- 
sion of operational procedures, decrease or increase in maintenance rates, and revision of 
measurement schemes to uncover sources of inadequate performance. The data would serve 
as a bas6 for the initiation of ECP's to remedy persistent failures. 

There will always be need for high-level technical expertise and judgment to maintain the 
performance of a complex system. However, if there exists an accessible, cumulative and 
continually updated record of system and subsystem performance (as well as operations and 
maintenance performance), then it is possible to exercise expert supervi'iion without an expert 
at each site or operational location. Advantage can be taken of modem communications. Prob- 
lems can be diagnosed and repairs effected over a telephone, and engineers can be flown in if 
necessary. 

This is essentially the mode of operation, for example, of large oil refineries. In this case, 
it is made possible by the existence of highly reliable, automated monitoring equipment with a 
central display of refinery operation. AF systems — radar, communication, command/control, 
avionic ~ almost always require human intervention for control, performance measurement, and 
maintenance. As a consequence, data are not generally gathered in machine readable form; they 
are not always reliable, and the aggregation of data for diagnostic purposes is more often a crisis 
operation than a routine procedure. 
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It is suggested that these deficiencies can be remedied by using the LTS for system per- 
formance assurance.*'' The LTS-4, which is currently under development, will be a completely 
self-contained and portable system suitable for use at remote field sites. It offers 



(a) An economical means for distributing, storing, and presenting job performance 
aids and guides for system checkout. In particular, it can take advantage of picto- 
rial procedures to insure accuracy and ease in the interpretation of instructions. 

(b) Automatic control and monitoring of maintenance procedures. The LTS incorpo- 
rates sophisticated decision processing,, i.e., it can process stored data and cur- 
rent information to select at each juncture the best succeeding procedure. By 
monitoring each step of the procedure, the system can eliminate typical human 
failures such as forgetting relevant data, omitting steps, or making erroneous 
decisions. 

(c) A point of operation, data entry device. By attaching a simple digital recorder^ 
to the LTS-4, it would be possible to record in machine readable form (1) the se- 
quence and timing of each operation performed by O/M personnel, (2) remedial 
actions taken, and (3) measurements and system performance data. This informa- 
tion can be processed for use by the site manager or by technical experts to eval* 
uate system performance, to anticipate (and avoid) system failure, and to evaluate 
the adequacy of maintenance procedures. 



Lastly, the system offers a significant capability for supporting on-the-job training. For 
instance^ it would be possible under technician or system control to branch to specific training 
(as opposed to performance aid) material. Exactly the same instruction could be available on- 
site as at the training school; it would, however, be available in a job connected context with 
visible relevance and the expectation of an enhanced motivation to learn. 



1. "Instructional System Development," AF Manual 50-2, Department of the 
Air Force (31 December 1970). 

2. R. C. Butman and F. C. Frick, "The Lincoln Training System: A Summary 
Report," Technical Note 1972-26, Lincoln Laboratory, M.LT. (3 October 
.1972), DDC AD-750748. 

3. C. D. Hunter, "New Dimensions to Instruction," USAF Instructors Journal 
(Fall 1970), pp. 11-15. 

4. J. P. FoUey, Jr., R. M. Woods, and J. P. Foley, Jr., "Comparison of Three 
Modes of Instruction for the Operation of a Complex Oscilloscope," 
AMRL-TR-66-195, W right -Patte rson AFB, Ohio (March 1967). 

5. "Fully Proceduralized Job Performance Aids," AFH:RL-TR-71-53, Vols. I 
and m, AFHRL (December 1971). 

6. R. L. Feik, AFCS/OA, personal communication. 

7. "Report Scores USAF on Navaids," Aviation Week and Space Technology 
(June 12, 1972). 



An illustrative example is given in Appendix II. 

t Alternatively, LTS-4 terminals can be directly connected to a central computer which would 
perform the necessary data reduction, storage, and display. 
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APPENDIX I 

COST ESTIMATES FOR PRODUCING PROCEDURALIZED AIDS 



Personnel costs represent the major factor in developing any form of proceduralized aid 
(PA). Table I presents time records of the personnel who developed the PA on calibration of a 
Tektronix 545 A oscilloscope. 

TABLE I 
PERSONNEL CHARGES 

Hours 



Instructor (laboratory technician) 14 

Producer (staff assistant) 30 

Instructional specialist 20 

Instructional programmer 4 

Photographer"^' (laboratory technician) 12 



Includes photography (4 hours), photo development, 
and art work. 

This was a first attempt at the development of a pictorialized procedure and therefore rep- 
resents only an approximation to times that might be required under commercial production 
conditions. In particular, a substantial fraction of the time spent by the producer and instruc- 
tional specialist was invested in learning, and this added substantially to the time required of 
the instructor/demonstrator. On the other hand, the material being developed was relatively 
simple in terms of content and the instructional strategy that was adopted. The total time in- 
volved - 80 man-hours for 30 minutes to 1 hour of material - is not out of line with that reported 
for the development of other forms of programmed instruction. 

The instructor and photographer are technicians, for which we will estimate $5 per hour; 
the producer and instructional programmer are staff assistants at $6,50 per hour; the instruc- 
tional specialist is a professional staff member estimated a' $12 per hour. To estimate the 
cost of commercial development, we add overhead (105%), G and A (10%) and profit (10%) to 
these figures to develop Table II. Cost of material" is low and assumed to be covered by 
overhead. 



TABLE n 
DEVELOPMENT COSTS 



Inst ructor 


14 hours 


@ 


$12.40 


$ 173.60 


Producer 


30 hours 


@ 


$16.13 


483.90 


Instructional specialist 


^ hours 


@ 


$29.77 


595.40 


Instructional programmer 


4 hours 


@ 


$16.30 


64.52 


Photographer 


12 hours 


@ 


$12.40 


148.80 










$1466.22 




The PA material is in the form of 8|- X 11 -inch typed text and continuous tone glossy prints. 
The audio messages are stored on magnetic tape and are converted to LTS formatted spiral 
records on 8|-- x 11 -inch film, using a special-purpose recording optical galvanometer. Tlie 
approximate costs, for time and materials, to process this material foi' microfiche distribution 
are shown in Table III. 



TABI.E III 
MICROFICHE PRODUCTION COSTS 

41 audio plates @ $1.25 $ 51.25 

41 video plates @ $1.25 51.25 

4 microfiche masters @ $4.50 18.00 

Duplicates @ $0.11 per card (4000 cards) 440.00 

Total cost 1000 copies $560.50 

Distribution cost, labor and postage $140.00 

Cost to distribute 1000 copies $700.50 

Cost per copy {four fiche) $ 0.70 



Xerox copies of this mal^*rial were made for the booklet used in the experiment, but we have 
developed approximate costs (Table IV) to produce suitably bound copies with good halftone 
quality photographs. Tht i me master PA material is furnished as input to this process. 



TABLE IV 
BOOKLET PRODUCTION COSTS 

35 ..-Ifcone plates @ $6.00 $ 210.00 

47 photo-direct plates @ $1.00 47.00 

Paper @ $4/1000 pages 164.00 

Printing @ $3/1000 impressions 246.00 
Binding 

Labor (collation, punching, binding) 400.00 

Bindings (1000 @ $0.10 ea) 100.00 



Total estimated cost $1167.00 

We discussed production of the booklet with several commercial printing shops, and ob- 
tained informal telephone quotations which average $1250 for 1000 bound booklets. This is to be 
compared with our estimate of $1167 for time and materials costs. Using the higher figure, the 
total estimated cost for 1000 booklets is $1250. 

Distribution, labor and postage (3rd class bulk rate) $ 200.00 



Total cost $1450.00 

Cost per booklet $ 1.45 

While the above cost estimates are only approximate, we believe that the general conclusion 
is accurate — audio-graphic microfiche (12 frames per card) can be produced and distributed at 
substantially less cost than conventional printing and publishing of the same material. 
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This conclusion holds even if we were to use line drawings, rather than photographs 
{cf. JPA-UH-IH-2TS-1 7). In this case, the time required of the instructor/demonstrator and 
pi lor.og raphe r would be reduced by about two -thirds and halftone plates would not be required in 
production. In the present example, this might represent a savings of $400 or about 147c. How- 
ever, this is outweighed by the cost of producing the drawings. 

If we assume that the ratio of separate drawings to instructional frames is only 1:3, we 
would still require 11 drawings, which we estimate at appi'oximately $64 each (4 hours X $16 per 
hour for artist-illustrator) for a total cost, exclusive of assembly and layout, of approximately 
$700. 

A final consideration in comparing cosls between conventional publication and microfiche 
involves the cost of revision. As pointed out in the body of the report, we can anticipate a need 
for the revision of any training or performance aid material after initial tryouts with trainees. 
In the present case, we have estimated that approximately 20% of the material would be helped 
by revision. This would be increased to 50% if we wished to take advantage of the memory and 
recording capability of the I^TS as suggested in Section IV, (This would not be a consideration 
for the booklet. ) 

Development charges for revision are independent of the publication medium. Production 
charges for revised microfiche wo^ld run approximately $68 (for 50% revision). This is derived 
from Table III as follows: 

8-20 frames revised $2.50 = $20-$50 
4 microfiche masters = $18 

There would be essentially no extra charge for duplication and distribution. Microfiche are 
duplicated and distributed "on demand," with no significant savings from large runs. In this 
situation, we would make only a few prints, try them out, and revise. Prints ot the final re- 
vised procedures would be made only to meet the actual demand. 

Small runs are not economical in conventional publication, and it would not be worthwhile 
to print a few booklets for test, evaluation, and revision prior to full production unless full 
production was a very large number. In the present case, major revision of the mp.terial, Z0% 
or more, would very nearly duplicate the costs presented in Table IV. 
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APPENDIX II 

SYSTEM PERFORMANCE ASSURANCE: AN ILLUSTRATION 

A simple example may give a better understanding of how the system suggested here might 
work in practice. 

The System Performance Assurance (SPA) concept involves the development of a set of- 
pictorial procedures for use with a (perhaps modified) LTS-4. These procedures will, in most 
cases, closely resemble acceptance test procedures and might be developed by the same engi- 
neering group. In any event, the intent would be to guide a technician through system setup, 
appropriate performance measurement and normalization, or component replacement where 
required. Most of the actions to be performed will use mode-switch options and observations 
of built-in instrumentation; readings for these instruments will be recorded through the LTS 
keyboard. They will be u.:-.irj to determine the O/M procedure sequence, to monitor O/M per- 
sonnel performance, and to update system performance records. 

Typically, thejse procedures will involve a number of conditional branch points such as the 
following»where the O/M man must make a power measurement and enter the result. Suppose: 

greater than +3 dBm high gain fault 

-fl — +3 dBm marginal 

—1 - +1 dBm normal 

—3 1 dBm marginal 

less than —3 dBm low gain fault 

The design engineer will have programmed the LTS for subsequent action at this point. If 
he considers the observed value to represent a system fault, he will arrange a branch to a fault 
search or fault clear procedure. When this procedure is completed - a module is replaced or 
an adjustment is made which clears the fault — the system displays an initialization frame for 
resetting the proper mode of operation, and the procedure continues to the next main-line frame 
(which is also the "normal" branch). If a high normal or low normal response was entered, a 
weighted number is stored for use in calculating a "system quality" factor. Based on the cumu- 
lative total of high or low normal measurements, the operator may be branched to a tune-up or 
recalibration procedure, or he may be continued in the normal checkout mode. 

A recording would be made of all the performance measures entered and the times and the 
sequence of actions taken (branches). Relatively simple automatic data processing would then 
provide continuously updated information on the maintenance history as well as module, sub- 
system^ and system performance. These records, interpreted by a trained system engineer, 
could provide the basis for preventive maintenance and the generation of ECP's in addition to 
the usual development of logistical support needs. 

In order to evaluate these general concepts^ we have examined them against the ALT AIR 
radar system. The ALTAIR radar is a modern high technology system; it is highly modular; 
it has considerable designed-in test equipment; and it provides for monitoring of subsystem per- 
formance while the system is on-line and operating. The specifications for procurement of this 
radar yystem were generated in 1963-1964 (almost a decade ago), and we believe that it is rep- 
resentative of much of the present-day military radar and communications systems, in-place 
and operating globally. Aside from the high power transmitting TWT's, klystron and vacuum 
tubes, the system is entirely solid state. It is compartmentalized into a set of major systems, 
each operated and monitored from its own test console. 
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The Calibration and Monitoring subsystem (C/M) is the on-line equipment which is used to 
perform power, gain^ phase, and noise figure measurements on portions of the receiver, exciter, 
and transmitter subgroups. When the ALTAIR system is in a mission, the C/M is used for mon- 
itoring purposes only. Receiver, exciter, and transmitter power monitor functions are con- 
trolled by the Systeni Data P^.stribution and Timing subgroup (DDT) via the C/M control unit. 
When the ALTAIR system is not in a mission, then the C/M may be used for either calibration 
or monitoring purposes. Under this condition, either the DDT, the C/M, or a combinatioa of 
both will control the functions of the subgroups. 

All the above-mentioned measurements may be performed in the receiver by using the ex- 
citer, a noise generator contained within the receiver rack, or the actual received signal as a 
test signal. The RF level to the receiver is adjusted and passed to the appropriate receiver 
front end. When the noise generator is used as the signal source, noise is injected directly 
into the receiver front end. In any case, the desired IF receiver output is selected by the dis- 
tribution system and sent to the C/M for measurement. The output of the C/M then appears on 
either a power meter or on an oscilloscope located on the C/M front panel. The exciter HF out- 
put may be monitored on the power meter or displayed as detected video on the oscilloscope. In 
this mode of operation, the selected exciter output is passed directly to the C/M display device 
and not to the receiver. 

The personnel responsible for this subsystem, as well as the systems which it monitors, 
use this facility for 



(a) Daily tests for gross characteristics such as gain, S/N, frequency, sidelobe level, 
etc., 

(b) On-line (operational) monitoring of the system on a non-interference basis, to 
detect and act on fault alarms, 

(c) When radar system test has isolated a fault in this system, and it is required that 
it be found and cleared, 

(d) Periodic (but infrequent) system test verifying that the system meets design per- 
formance specifications. 



A section of Receiver Calibration Procedures is included (Appendix III) to show the format 
of the textual material which is provided to support maintenance personnel in performing these 
functions. Tt should be noted that while the ALTAIR manual has been highly proceduralized, the 
procedures are "success oriented" in that there is no provision for other than nominal operation, 
and highly skilled maintenance personnel must be imniediately available to trouble shoot when 
faults occur. The many references to tables and the need to perform frequent calculations add 
significantly to operator error and unnecessary adjustment and replacement of equipment mod- 
ules. A computerized maintenance support system, such as we are proposing, would eliminate 
this source of system performance variability. 

However, the most iraportant contribution to system performance assurance d'=?rives from 
the capability of the LTS to monitor the performance of O/M personnel (just as it .nonitors the 
performance of students in a training situation). Not only is it able to control and record the 
sequence of checks aJid operations performed, but it is able to store the results and modify pro- 
cedures on the basis of prior tests and measures. 




ERIC 



14 



For instance, there may be as many as twenty modules consisting of R"F amplifiers, digitally 
controlled attenuators, IF amplifiers, cables, coaxial mode switches, etc., in the chain over 
which it is desired to measure system gain. 

In a typical procedure, the operator will enter a CW power measurement at a number of 
node points (see Fig. 1): 

(a) Output of front-end amplifier, 

(b) Output of mixer-preamplifier plus cable run to console, 

(c) Output of front IF amplifier, 

(d) Output of second IF amplifier following mode switches and hybrid power dividers, 
attenuators, etc. 

At any point, an abnormal gain or loss will cause a branch to a fault location and clear procedure 
for the module under test. The nominal power is reset at each node by adjusting the level of the 
input test signal, and ;.: 3 new level is entered via the keyboard. A cumulative too low/too high 
gain condition would be detected in this way. It is also possible to measure nominal over-all 
path gain while intermediate modules are operating at abnormal gain. There might be too low 
front-end amplifier gain which is compensated by too high IF amplifier gain. By eliciting re- 
sponses from the operator in a logical sequence, it is possible to detect abnormal conditions 
and instruct the operator in restoring the module(s) to nominal operation. The operator might 
continue through most of the modules .n this path without encountering a fault, but the "quality 
factor" (number of modules with acceptable but marginal performance) may exceed a clip level 
(too many marginal modules), and the operator would be branched to the start of the sequence. 
At this point, he would be required to restore each module to nominal performance. 

Using this approach, the author of the SPA maintains control of the operating characteristics 
of the system. Modules are allowed to drift within designer specified limits without continual 
trimming. Rates of drift are determined and an optimum required rate of O/m can be established. 
Defective modules (based on failure rate, drift rate, magnitude of drift, etc.) are identified, and 
replacement or ECP's can be initiated. 

Decisions relative to nominal values and acceptable margins are thus made by the system 
expert, not by O/M personnel. The SPA system remembers modes, calibration constraints, 
logical routes to task completion, etc., and keeps records in a format that permits automatic 
data processing for performance accountability of both the system and the operator. 
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APPENDIX III 
RECEIVER CALIBRATION PROCEDURES 



A section of the general procedure for performing gain measurements has been taken from 
the ALTAIR Radar System Calibration and Monitoring System (CAL/MON) Operation and Main- 
tenance (O/M) Manual to illustrate the state-of-the-art approach in supporting field O/M per- 
sonnel. A typical set of log sheets from the CAL/MON O/M Manual is included to illustrate the 
number of node point measurements invoU ed in receiver gain checkout. 
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4.3.2.1 General Procedure 

The general pr ^edure for making the above measurements is as follows: 

1. Establish the proper input test signal levels for the selected Receiver channel. 

2. TJ^stablish the Receiver IF output levels which are to be monitored and measured. 

3. Select the monitoring device. 

4. Make the d 't^irec measurements. 

The positions for significant Control Panel controls for each type of measurement are 
given in Table 4-2. The following procedures presume that these controls have been properly 
positioned prior to performing the calibration measurements. 

4.3.2.2 Gain Measurements 

In this subsection, two different procedures for making gain measurements are given: 

Signal Method of Measurement 

CW " Power Meter 

Pulse Scope/Power Meter 

To m.^ke gain measurements in the GW mode, the power meter is used to set up both the 
Receiver input and output levels. 

CW Gain Measurement - Power Meter - The procedure for making CW gain measurements is 
given in Table 4-4. 



TABLE 4-4 
PROCEDURE FOR CW GAIN MEASUREMENT 


Step 


Procedure 


1 


Position the C/M switches for Mode ?. as shown in Table 4-2. 


2 


Rotate the CHANNEL SELECT switch to the desired UHF or VHF channel. 


3 


Rotate the OUTPUT SELECT switch to the desired Receiver output. 


4 


Select the desired waveform {VCW or UCW) with the WAVEFORM SELECT switch. 



ERIC 



18 



TABLE 4-4 (Continued) 
PROCEDURE FOR CW GAIN MEASUREMENT 



Step 



Procedure 



Place the DISPLAY SELECT switch in the desired POWER METER position 
(VHF IN or UHF IN). 

With the precision step attenuator set at 0 db, monitor the exciter output on the 
power meter and record the reading. 

From Table 4-5 obtain the calculated loss factor between the C/M and the 
Receiver front-end for the selected channel. 

From Table 4-6 obtain the suggested Receiver input level for the selected 

Receiver channel. This is Power. . 

m 

Calculate the input attenuation from the following formula: 
ATT = 0 db power (Step 6) - Losses (Step 7) - Signal Level (Step 8) 

10 Set the precision step attenuator to the value obtained in Step 9. 

NOTE 

With the precision step attenuator in the 
required position for the selected channel, 
the input level to the Receiver is 10 db below 
saturation. 

To establish the proper IF output level for the selected Receiver channel, continue 
below. 

11 From Table 4-7, obtain the IF attenuator setting for the selected Receiver channel. 

12 On the IF attenuator panel, depress the CONTINUOUS/STEP switch-indicator so 
that it reads STEP. 

13 Set the IF step attenuator to the value obtained in Step 11.. 

14 Place the DISPLAY SELECT switch in the IF OUT position. 

15 Observe the power meter reading and record. 

16 From Step 8 obtain the input power, and from Step 15 obtain the output power. 
Calculate the Receiver gain from the following formula: 

Gain = Power , — Power. . 

out in 
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TABLE 4-5 
RF INSERTION LOSS 



Channel 


VHF 


SR 


VHP 


2L 


VHF 


AAZ 


VHF 


AEL 


VHF 


2R 


VHF 


2L 


VHF 


AAZ 


VHF 


AEL 


VHF 


2R 


^ VHF 


2L 


VHF 


AAZ 


VHF 


AEL 


UHF 


2R 


UHF 


SL 


UHF 


SR 


UHF 


2L 


UHF 


2R 


UHF 


2L 



Frequency(lVIH2) 



153 

153 

153 

153 

157.5 

157.5 

157.5 

157-5 

162 

162 

162 

162 

415 

415 

427 

427 

440 

440 



Loss(db) 



13.75 
13.75 
13.85 
13.90 
13.85 
13.85 
13.95 
14.00 
13.90 



14.00 



14-10 
14.10 
14.15 
14.10 
14.35 
14.35 
14.50 
14.45 
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TABLE 4-6 
RECEIVER INPUT LEVELS 











Suggested Input Level'^' 
Waveform 










CW 


S 


L 


UHF 


HIGH 


GAIN 


(1) 


-79 dbm 


-93 dbm 




-100 dbm 




UHF 


MED 


GAIN 


(2) 


-55 dbm 


-69 dbm 




-76 dbm 




UHF 


LOW 


GAIN 


(3) 


-31 dbm 


-45 dbm 




-52 dbm 




VHF 


HIGH 


GAIN 


(1) 


-84 dbm 


-97 dbm 




-104 dbm 




VHF 


MED 


GAIN 


(2) 


-60 dbm 


-73 dbm 




-80 dbm 




VHF 


LOW 


GAIN 


(3) 


-36 dbm 


-49 dbm 




-56 dbm 





^"All input levels are 12 db below saturation. 
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TABLE 4-7 



IF ATTENUATOR SETTINGS 





Channel 


Attenuation 




V2R1 


31.20 




V2R2 


31.40 




V2R3 


31.50 








31.35 






V2L2 




31.35 






V2L3 


31.45 




V2L TRK 


27.25 




V2L RGC 


27.50 




VAAZl 


31.60 




VAAZ2 


31.40 




VAAZ3 


31.45 




VAAZ TRK 


27.40 




VAELl 


31.45 




VAEL2 


31.55 




VAEL3 


31.40 




VAL l.HK 


27.25 




U2R1 


31.45 




U2R2 


31.35 




U2R3 


31.10 




TJ2L1 


31.55 




U2L2 


31.50 




U2L3 


31.80 




U2L TRK 


29.40 




U2L ROC 


28.10 
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APPENDIX A 
VHF RECEIVER GAIN CALIBRATION 

Date 

Time 

A-1. RECEIVER GAIN 

1. EXCITER dbm with 0 db RF ATTENUATOR 

2. C/M dbm with 30.27 db in CONT ATTENUATOR 

(adjust for +9.0 dbm) 

48.8 db IF STEP ATTENUATOR 

3. RF DISTRIBUTION LOSSES at 162 MC: R = 13.90 AZ = 14.10 

L = 14.00 EL = 14.10 

4. INPUT -36 dbm (RF ATT. 22.0 db + ITEM 1 = ) 

CHANNEL IF ATT. OUTPUT LEVEL EXP ECTED 

LEVEL +0.25 db 



TGC = 63 db SR TRK (RGC) 


27.65 


dbm 


+ 8.0 


SL TRK 


27.20 


dbm 


+10,0 


AZ TRK 


27.45* 


dbm 


+10.0 


EL TRK 


27.40* 


dbm 


+10.0 


£R 3 (Recorder) 


32.45** 


dbm 


- 2.0 


2L 3 (Recorder) 


32.50 


dbm 


- 2.0 


AZ 3 (Recorder) 


32.55* 


dbm 


- 2.0 


EL 3 (Recorder) 


32.70* 


dbm 


- 2.0 


5. INPUT -60 dbm (RF ATT. 


ITEM 4 + 24 db = 




) 


TGC - 37 db SL TRK 


27,20 


dbm 


+10.0 


AZ TRK 


27.45* 


dbm 


+10.0 


EL TRK 


27.40* 


dbm 


+10.0 


6. INPUT -84 dbm (RF ATT. 


ITEM 5 + 24 db = 




) 


TGC = 15 db TRK 


27.20 


dbm 


+10.0 


AZ TRK 


27.45* 


dbm 


+10.0 


EL TRK 


27.40* 


dbm 


+10,0 



7. IIECHECK EXCITER: ^dbm, and C/M: dbm 

8. COMPARE EXCITER VCW to CW (ref): ^= Al 

* 0.1 db added to correct for RF distribution error. 

* 0.1 db substracted to correct for RF distribution error. 

COMPARE EXCITER VL to VCW (ref): = A2 

COMPARE EXCITER VS to VCW (ref): = A3 

9. RECEIVER PULSE GAIN (compare VL ^ VS to VCW) (MAX ALLOWABLE 
ERROR ±0.5 db) 

VCW input -36 dbm (RF ATT. = same as ITEM 4 = ) 

VL input -56 (RF ATT. = ITEM 4 + 20 db + A2 = ) 

VS input -49 dbm (RF ATT. ITEM 4 + 13 db + 3 - ) 

2R 2L AZ EL 

VL db db db db 

VS db db db db 

A-2. RECEIVER PHASE 

1. C/M (Cont. IF ATT. = 34.0 db) 

CW 2. Input ^ -36 dbm £L M EL ATT. CHECK 

(RF ATT. = 20-3-3.1) 

TGC = 60 db RECORDER (3) (6.5) 



TRK (17) _ 

3. Input ^-60 dbm 

(RF ATT. = 40-7-3.1)TRK (17) 

TGC = 36 db 

4. Input ^ -84 dbm 

(RF ATT. = 70-l-3.1)TRK (17)_ 

TGC = 12 db 

VCW 5. Input ^ -36 dbm RECORDER (3) (6.5) 

(RF ATT. = 20-3-3.1) 

TGC = 60 TRK (17) _ 

VL 6. Input ^ -66 dbm RECORDER (3) <6.5) 

(RF ATT. = 40-3-3.1) 

TGC = 60 TRK {17) _ 

VS 7, Input ^-49 dbm RECORDER (5) (6.5) 

(RF ATT. = 30-6-3.1) 

TGC = 60 TRK (17) 



Max. Allowfible Difference between SL and AZ or EL is * 5 degrees. 
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